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SIJi...MARY 



This paper summarizes twenty-two recent, empirical studies on the cost- 
effectiveness of military training icported by countries that participate in The Technical 
Cooperation Program. The studies show an almost exclusive interest in simulation that 
extends from full-task, highly realistic simulators to part-task, lower cost simulators. The 
data show that simulator-based training is effective in developing performance proficiency; 
some transfer fo>m simulator-based training to performance on actual equipment occurred 
in every case. Speed of performance was initially slower on maintenance tasl^ for 
simulator trained students than for those trained with actual equipment. As expected, 
simulatCK^ training was found to cost less than actual equipment training. Simulator training 
was found to be more cost-effective than training with actual equipment llie types of cost 
and effectiveness data collected m these studies do not permit us to examine trade-offs 
between cost and effectiveness. Future studies of training should examine functional 
relationships between effectiveness and cost and include sensitivity analyses of key 
variables that determine the cost-effectiveness of alternative training ^proaches. 
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I. INTRODUCTION 



This paper summarizes recent studies on the cost-effectiveness of training 
performed in member countries of The Technical Cooperation Program (TTCP): Australia, 
Canada, New Zealand, United Kingdom, and United States. It is intended to update an 
earlier TTCP report on the same topic (Chatclier, Harvey, and Orlansky, 1982). The 
present studies, limited primarily to analytic and/or empirical comparisons of new with 
existing methods of trainirig. were provided by members of the Technical Panel on 
Training Technology (UTP-2), TTCP. This paper discusses issues, iata, and conclusions 
presented in the studies. It is not a complete review of all cost-effectiveness studies that 
might be found by a more comprehensive search in the five TTCP countries. 

This paper includes the following: 

(1) Discussion of the methodology used in this paper to summarize the empirical 
studies; 

(2) Results summarized from the studies; 

(3) Discussion of the studies; 

(4) Conclusions regarding the effectiveness, costs, and cost-etfectiveness of the 
types of training that were investigate^*; 

(5) Reconmiendations based on these studies. 
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11. METHODOLOGY 

This paper considers 27 reports that are listed in the References, Twenty-two 
empirical studies of training were found in i3 of ther^e reports (summarized in Table !)• 
Seven acceptance tests and analyses of training are described in six reports (summarized in 
Table 2). Five surveys and analyses of training arc described in five reports (summarized 
in Table 3). Finally, survey methodology for evaluating training is discussed by three 
reports (sammarized in Table 4), The reference list and Tables 1 through 5 are presented 
following the main text 

Alternative approaches to training were compared and evaluated in each of the 22 
empirical studies shown in Table 1. All of the studies examined some type of simulation 
for use in military training. The measures of effectiveness and of costs used in these 
studies are also presented in Table L Quantifiable findings, whenever they were available 
or derivable, are also snown in this table. 

Three ratios, described below, are used wherever possible to report the findings of 
these studies: 

Transfer Effectiveness Ratio (TER) 

Training Cost Ratio (TCR) 

Cost-Effectiven'^s Ratio (CER). 

In some cases, the ratios were provided by the studies. In other cases, we used 
data provided by the studies to calculate the ratios. In several cases, we did not have 
enough data to calculate any of these i "itios* 

The Transfer EfTectiveness Ratio (TER) is the ratio of training time saved by use of 
simulation to the amount of time spent in simulation: 



where, 



S 



A = time required to reach criterion performance using actual equipment 
without prior use of a simulator. 
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Ag = time required to reach criterion performance using actual equipment with 
prior use of a simulator, 

S = time spent training in the simulator. 

In some studies (shown in Table 1), numbers of training trials are used as 
indicators rathu than houn of training time. 

The TER permits us to estimate the value added to a training program by the use of 
simulation. It is not suitable for all training programs. For instance, some training 
programs will not use actual equipment for reasons of economy or safety. Positive TERs 
indicate positive transfer of training, i.e., a savings in actual equipment time attributable to 
time spent in simulator ♦raining. Therefore, the larger the positive value of a TER, the 
greater the value added by simulation to training. However, how "good" a given TER 
value is depends on how much less expensive the simulator training is than the actual 
equipment training. The ratio of these costs is given by the Training Cost Ratio. 

The Training Cost Ratio (TCR) provides a standardized measure of tiie amount of 
money saved by one (usually new, simulator based) versus another (usually existing, 
actual equipment based) training program or s^roach. It is defined as: 

TCR = ^ 
where, 

Cs = cost of the new training approach, 
C = cost of the existing training approach. 

'Cost' must be defined. Often, it is the variable hourly operating cost of the 
simulator and the actual equipment In Table 1 , the cost bases of each TCR are noted in 
the ^Measures of Cost' column. All TCRs reported in the table are based on hourly 
operating costs of simulators and actual equipment In these cases, smaller TCRs favor 
simulator ti dining* If the TCR is less than 1.0, it costs less to use simulators than actual 
equipment 

Variable operating costs refer to wh?t it costs to use training equipment Any costs 
saved by using a new sihiulator can be used to estimate how long it would take to recover 
spent funds, i*e., the amortization period. After this period, continued use offers absolute 
savings. These savings should be considered in light of all the costs associated with a 
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method of training, i.e., research and dev dopment, initial investment, and operating and 
sui^port costs. 

Given standardized ratios of effectiveness and cosis, we can turn to a st3ndardized 
ratio of costs to effectiveness, ihe Cc^t-Effectiveness Ratio (CER) is defined as follows: 

CER = ^ 
TER 

where, 

TCR = the Training Cost Ratio, 

TER = the Training Effectiveness Ratio. 

Defmed in this way, the closer the CER is to zc-^., the more cost effective the 
simulator-based training. Unde this defmition there ace no positive or negative theoxet^cal 
limits to CERs, and it is impossible to tell from a single CER if some threshold for cost- 
effectiveness has been crossed 

In sunmiary, a cost-effective training approach should have a large Transfer 
Effectiveness Ratio, a small Training Cost Ratio, and a snutll Cost-Effectiveness Ratio. In 
keeping with common notions of quality, it is occasionally suggested that the Cost- 
Effectiveness Ratio be reversed and made an Effectiveness-Cost Ratio so that larger ratios 
indicate greater value. This is a reasonable suggestion, but the usual practice is to measure 
and report cost-effectiveness. That approach is used in this paper. 

These ratios are essential components of cost-effectiveness studies. However, only 
three studies in Table 1 report CERs. All three of these studies were completed for the 
United Kingdom Armed Forces. 

The 22 sridies listed in Table 1 include nine on flight training, seven on 
maintenance training, and six onjtaining for some type of military operation (called 
"operations training"). Three of the operations training studies concerned gunnery training. 
Three of the 22 studies were from Canada, seven from the United Kingdom, and 12 from 
the United States. 

In the earlier TTCP report, Chatelier et al. (1982) presented a framework for 
estimating the cost and effectiveness of training devices and procedures. They also listed 
the categories of data needed from TTCP countries to expand on results already available. 
These categories are listed in Table 5 along with the number of studies from Table 1 that 
provide data for thent 
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in. RESULTS 

The use of sunulation for training, documented by the 22 studies listed in Table 1, 
appears to be effective. The Transfer Effectiveness Ratios range from ,67 to ,99 (Brearley, 
1980). These ratios exceed the 50 percent median time savings repented by Chatelier et al. 
(1982) and shown in Table 3. 

Some transfer from simulator training to actual equipment perfonnance occurred in 
every case-although performance on acti'^al equipment was not always superior among 
simulator trained students. There were 16 stjdies that conq[>arcd simulator trained students 
with students who were trained using actual equipment Among these studies, four 
reported superior performance proficiency among simulator trained students, eight reported 
equal perfonnance, and four reported infericx' performance. No studies reported a complete 
absence of transfer. These results are consistent with those repealed in Table 3 by Chatelier 
et al. (1982) who concluded that simulators and actual equipment were generally equally 
effective for training. 

Simulator training appears to reduce speed of poformance with actual equipment, 
compared to training only with actual equipment Seven of the TTCP studies included data 
on the time students needed to perform a given task on actual equipment All of these 
studies concerned electronic or automotive maintenance. Six of these studies reported that 
students trained with simulators performed significantly slower on assigned criterion tasks 
than did students trained with actual equipment This result was not anticipated by any of 
the surveys and atudyses listed in Table 3. 

Not surpricmgly, these TTCP studies indicate that simulator training costs less than 
training using actual equipment The five 1*CRs based on hourly operating costs range 
from .11 to .50 (see Brearley, 1980, in Table 1). These TCRs are all for aircraft and 
aircraft simulators used in pilot training. They are generally larger than the .12 median 
TCR found by Chatelier et al. (1982) but within the range of .02 to .59 found more 
recently by Orlansky et al. (1984) and reported as a median value of .08 in Table 3. They 
arc well below the value of 1.00 that would obtain if the costs of siiaulation and actual 
equipment training were the same. The five TCRs based on acquisition costs range from 
.008 (Court and Sharrock, 1985) to 2.60 (Cicchinelli, Keller, and Harmon, 1984). The 
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low value of .008 represents the ratio of the cost of a small computer system to a large 
sonar system. It does not include the cost of software required for simulating the operation 
of the sonar. It is a good argument for the use of a part-task trainer. The high TCR of 
2.60 is based on initial acquisition of a single trainer. Acquisition of subsequent copies of 
the trainer reduce the TCR to .40. Two TCRs are based on life-cycle costs. They are 1.00 
(Cicchinelli et al., 1984) and .03 (Pieper, Richardson, Harmon, Keller, and Massey, 
1984). 

Training using simulatioa was found to be cost-effective. The four CERs reported 
in Table I range from .11 to .74 (Brearley, 1980). These CERs are again for aircraft 
simulators used in pilot training. They are slightly lower than the results from similar 
studies that were reported by Chatelier et al. (1982), where the CERs range from .24 to 
1.30 depending on what measures of cost are used (see Table 3). 
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IV. DISCUSSION 
Seven general observations may be made about these 22 empirical studies. They 
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First, these studies show zn overwhelming interest in the use of simulation for 
training. The original request for studies collected for this paper was only for comparisons 
of new with existing methods of training. There was no requirement for any of these 
studies to deal with simulation. Yet every one of the 22 empirical studies received and 
summarized here concerns simulation-based training. 

Second, interest in simulation extends from the use of full task, highly realistic 
simulators to the use of part-task, lower cost simulators; both extremes are viewed as 
alternatives to training, at least in part, with actual equipment Sayer's (1984) informal 
survey (Table 3) turned up 47 low cost, part-task trainers in the United Kingdom, New 
Zealand, and Australia (he did not receive responses firom Canada and the United States). 
These studies show that the field of training has progressed beyond the notion that 
simulation is the province cmly of high cost aircraft simulators. Moreover, the basic 
assumption that we must have fuU-task simulators has been questioned. For instance, 
Waag*s (1981) review (Table 3) suggests that motion platforms, one of the most costly 
components of aircraft simulators, do not improve peifomuuice or save flight time. As we 
evaluate and eliminate various components that heretofore seemed essential to our training 
programs, the issue of cost-effectiveness becomes increasingly important We need to 
know how much a cost savings in training reduces its effectiveness— if at all. Alternatively, 
we need to know how much increased resources for training will buy in terms of increased 
effectiveness. 

Third, the range of cost data provided by these studies remains narrow. As Table 5 
shows, only four of ten possible cost categories were considered in this sampling of 
studies; this can only mean that the cost data are incomplete (and potentially misleading). 
For that matter, 14 of the 22 studies listed in Table 1 did not provide any cost data. Of the 
remaining studies, three reported iiourly operating costs for simulators and actual 
equipment, one reported annual operating costs, three reported acquisition costs, and two 
repented life-cycle costs. Although student cost savings were not considered by any of the 
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studies, student time savings were. Eight of these time savings measures were directly 
related to student costs. In these; studies a direct attempt was made to measure the amount 
of time students required to reach training criteria using actual equipment In all of these 
cases, the 'actual equipment* was an aircraft. In the six acceptance testing and analysis 
studies listed in Table 2, the range of cost categories was only slightly wider. Five studies 
considered acquisition costs, three considered student time to reach criterion, three 
considered development costs (usually measured as conversion or transition costs), two 
considered operating costs (in one of these cases operating costs were measured by 
ammunition costs), and one ccmsideied instructor costs (number of instructors). 

Fourth, the range of effectiveness measures considered by each individual study 
remains narrow. No study considered all five of the effectiveness categories listed by 
Chatelier et al. (1982). Only seven studies considered attitudes of either students or 
instructors, only three considered student attrition, and none considered on-the-job 
performance. On the other hand, all of the studies considered end-of-course effectiveness, 
and all but two of the studies considered transfer of training from simulators to real 
equipment Student time savings in use of actual equipment was treated as both a cost and 
an effectiveness measure: eight studies measured time to complete a task using actual 
equipment; four other studies measured success in maintaining or operating actual 
equipment as an indicator cS effectiveness. 

Fifth, the emphasis in these studies is on simulation as an inexpensive form of 
training rather than as a source of unique training capabilities. Despite our extension of 
simulation technology beyond the full-task, high fidelity simulators used in pilot training, 
simulation is still viewed as a substitute for the actual equipment training we would prefer 
to have. Transfer effectiveness, with its focus on training time saved in actual equipment 
usage, typiHes this notion. Simulation provides capabilities for lts% costly and safer 
training. These capabilities are fairly well captured by notions of transfer and cost- 
effectiveness. However, simulation also provides levels of feedback on performance, 
control over the difficulty and sequencing of instructional events, and replay capabilities 
that are unattainaUe fm most actual equipment used for training. These capabilities were 
infrequently considered in the TTCP studies. 

Sixth, nearly all the studies were concerned with high technology 2Q)proaches to 
training. The four studies that were exceptions concerned slide-based simulation. 
However, they involved one degree or another of interaction, i.e., the slide-based simulator 
reacted in some veridical fashion to the actions and decisions of students. Low technology 
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still has a role in training. Cost-effectiveness studies should help determine what that role 
is. 

Seventh, just as there are effectiveness studies that neglect cost considerations, 
there are acceptance studies that neglect effectiveness considerations. Fourteen of the 
empirical studies listed in Table 1 did not consider any measures of cost, and three of the 7 
acceptance studies listed in Table 2 did not consider any measurca of effectiveness. 
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V. CONCLUSIONS 



1. Training using simulation is effective, based on data contained in the TTCP 
studies. All the TERs reported were for aircraft simulators. The savings in flight time 
were about 40 minutes for every hour spent in a simulator. There was great variability in 
the size and type of performance improvements reported; the average percentage 
improvement among aU measures of effectiveness was about eight percent 

2. Some transfer of performance firom simulator training to actual equipment 
always occurred; no negative TERs were reported. However, four studies found that 
simulator trained students performed less well on criterion tasks than did students trained 
on actual equipment; diese studies did not rep(»t TERs. An intexpretation of these findings 
that simulator trained students would perform less well than a control group that received 
no training should be tested 

3 . Performance speed in maintenance tasks on actual equipment is initially slower 
among students who were trained widi simulators than among those who were trained with 
actual equipment. These findings all occurred in studies of maintenance simulation and 
indicated that simulator trained students took about 3 1 percent longer to perform criterion 
maintenance actions. This finding has not been reported previously. 

4. Simulator training costs less than trairiing that uses actual equipment Simulator 
training appears to reduce training costs by at least SO percent. Notably, however, the 
TTCP studies generally considered operating costs and did not include research, 
development, or initial investment costs. 

5. Training based on simulation is more cost-effective than training using only 
actual equipment The cost-effectiveness ratios reported in these studies indicated a savings 
of about SO percent in cost to obtain a unit of effectiveness. This conclusion is based 
primarily on operating costs. 

There are good reasons to be pleased with this sampling of studies. They show that 
although much remains to be done, there has been steady progress, and much has been 
accomplished. 
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VI. RECOMMENDATIONS 

These studies were p iformed in a variety of environments and for a variety of 
purposes. Even without a conunon strategy, however, they provide a perspective not 
otherwise attainable and lead to five recommendations. 

First, the range of cost data collected by evaluation studies should be extended. 
Some progress has been made because it is increasingly common to finu some cost data in 
training where none had been the rule. However, many effectiveness studies continue to 
neglect cost data, and the range of cost data that is reported is limited to onr or two 
categories that differ from study to study. 

Second, the range of effectiveness data collected by evaluation studies should be 
extended Considerable progress has been achieved in gathering end-of-course and 
transfer data More needs to be done to measure attrition, assess student and instructor 
attitudes, and link alternative training approaches with on-the-job measures of 
effectiveness. Studies, reviews, and analyses of training approaches, such as those listed 
in Table 2, have limited value if they neglect effectiveness data. 

Third, studies of the cost and effectiveness of various training approaches should 
not stop with these measures. We need descriptive information that explains the successes 
and failures of new approaches and prescriptive information that provides guidance to 
training designers. We need to know how and why these approaches achieve their ends as 
well as whether or not they succeed. 

Fourth, there should be agreed upon guidelines for minimum required cost data 
reported in studies that members of TTCP may wish to exchange for common use. A 
comprehensive treatment of research and development, initial investment, and operating 
and support cost categories, such as the treatment outlined by Knapp and Orlansky (1983), 
may exceed the budget, energies, and competencies of most training evaluators. However, 
some practicable guidelines might be devised that would generally suffice for training 
program designers. 

Fifth, there should be agreed upon guidelines for minimum required effectiveness 
data reported in studies that members of TTCP may wish to exchange for common use. 

11 



20 



Again, a comprehensive treatment of training outcomes may exceed the budget, energies, 
and competencies of training evaluators. However, some practicable guidelines might be 
devised for general practice. 
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Table 1. Empirical Studies of Training in TTCP Countries 



Type of 
Training 


Comparison 


Measures of Effect 


Measures of Cost 


Findings 


Study 


Rotary wing 

flight. 

U.SiArmy 


AH-1 aircraft training 
required with different 
numbers o! repetitions 
of Ct different 
maneuvers In AHLFS 
simulators (N - 34). 


Required repetitions to 
criterion in aircraft 


N/A 


Exponential model accounts 
for most savings in flight time 
due to prior use of simulator. 
Economically optimal mixes 
of simulator and flight time 
can be derived using the 
model and operating costs. 


Bicl^ley 
(1980) 


Fixed wing 

flight. 

UK/RAF 


JP5A Basic Flying 
Training before 
(N- 72) and after 
(N-66) introduction 
of simulator. 


Number of successful students. 
Required hours of dual flying. 
Required hours of solo flying. 
Total hours in simulator and 
aicraft combined. 
Student attitudes. 
Instructor attitudes. 
Academic test scores. 
Transfer Effectiveness Ratios. 
Training Cost Ratios. 
Cost par successful student. 


Hourly operating cost fcr 
simulator. 

Hourly operating cost for 
aircraft. 


No change in student output 
quality or quantity with use of 
simulator 
TER-0.g9 

Worst case 

TCR-0.50 

CER-0.50 
Best case 

TCR-O.II 

CER-0.11 


Breariey 
(1980) 


Same as 
above. 


JP3A Basic Flying 
Training before 
(N- 109) and after 
(N- 117) introduction 
of simulator 


Same as above. 


Same as above. 


No change in student output 
quality or quantity with use of 
simulator 
TER-0.67 

Worst case 

TCR-0.50 

CER-0.74 
Best case 

TCR-0.11 

errs -0 17 


Same as 
above. 


Aircraft 
recognitton 
and identi- 
fication. 
UK/RAF 


Training using slides 
versus videotape 
presentation (N » 26). 


Number of aircraft correctly 
identified. 

Number of aircraft conrectly 
recognized. 


N/A 


Equal recognition and 
identification accuracy for 
slide and video presentations. 


Burnett 
(1983) 



Table 1. Empirical Studies of Training in TTCP Countries (continued) 



Type of 

Training 


Comparison 


Measures of Effect 


Measures of Cost 


Findings 


Study 


Electronic 

naintenance. 

U.S.AJSAF 


Training using simulator 
(2D-EEklT)(N-100) 
versus adual aauio* 

VHmIJMI 

mant (AN/SPS-10) . 
(N-47). 


Troubleshooting accuracy on 
actual equipment 
Troubleshotina soeed on actual 
equipment. 
Student r^ttltudes. 
Instructor attitudes. 


.\oquisitk)n and life cycle 
costs of simulator. 
AcQuisition and life cvcle 
costs of actual equipment. 


Equal troubleshooting accuracy 
for simulator and actual equip- 
ment groups. 

Troubleshooting speed 6-31% 
sbwer for simulator groups. 
TCR - 2.60 (acquisftton of 1st 
copy) 

TCR . n in /attiuisitlon of 2nd 
copy) 

TCR- 1.00 (lifa-cycia) 


Ck:chinelli, 
Keller. & 
Harmon 
(1984) 


Radar Opera^ 

uon fur r\n 

Traffic Control. 
U.S./RAF 


Trsining with and 
wiuioul simuiation. 


Successful operatton of actual 
equipmeni. 


N/A 


Performanc* of groups with 
Simulators oppareniiy superior. 


Courts 

onarrocK 

(1985) 


Sonar Opera- 
tion. 
UK/F)AF 


Training using part-tasl( 
versus full mission 
simulator. 


Successful operatton of actual 
equipment. 


Acquisrtton costs of part-task 
simulators. 

Acquisitk>n costs of full 
misston simulators. 


Performance of part task and 
fun misston groups about equal. 
TCR -0.01 


Same as 
above. 


Rotary wing 
instrument 
flight. 
UK/Army 


Training using Lynx 
simulator (N - 3) versus 
Lynx aircraft training 
(N.3). 


Instructor ratings in aircraft 
Hours to criterion in aircran. 
Transfer Effectiveness Ratk>s. 
Training Cost Ratios. 
Cost Effectiveness Ratbs. 


Hourly operating costs for 
simulator. 

Hourly ooerating costs for 
aircraft. 


Simulator training is as effective 

as aircraft training. 

TER-.-* 

TCR-...* 

CER-...* 


Fegetter & 

Ailnutt 

(1983) 


Air Support 

Flying. 

U.S.AJSAF 


Training with (N- 11) 
andwitf ,xjt(N-14) 
flight simulator for A-10 
surface attack skills. 


Sortie "survival" in ctose air 
support (RED FLAG) missbns 
in aircraft. 


N/A 


14% greater (survival in simulator 
group when simulator configured 
ck)sely to aircraft. 21% lower 
survival in simulator group when 
simulator configured less 
closely to aircri»ft. 


Hughes, 

Brooks, 

Graham, 

Sheen, & 

Dickens 

(1982) 



a 

Reported but restrk:ted information. 
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Table 1. Empirical Studies of Training in TTCP Countries (continued) 



Type of 
Training 


Comparison 


Measures of Effect 


Measures of Cost 


Findings 


Study 


Fixed wing 

Right. 

UK/RAF 


Basic Flight Training 
with and without 
Univarsity Air Squadron 
program. 


Required hours to critarbn in 
aircraft. 

Number of successful students. 


Annual program costs. 


Break-even occurs when - - 
ex-University Air Squadron 
students enter pitot training. 


Kilner & 
Elshaw 
(1984) 


Tank Gunnery. 


Training with (N- 6) and 
without (N - 6) gunnary 
simulator for Laopard 
battia tank. 


Average score on simulator. 
Hit time on simulator. 
Number of hits on simulator. 
Rounds required to achieve a hit 
on simulator. 

Number of 1st rounvi hits on 

oililUlalUi . 


N/A 


Average score improved 90% 
over ten 16'missk>n sessbns. 
Time to achieve hit reduced 21%. 
No significant increase in any 
accuracy measure. 


Magee 
(1984a) 


Tank Gunnary. 
Canadian Army 


Training using gunnaiy 
simulator (N- 20) 
varsus oonvantional 
instnjctkx) (N- 16) for 
Laopard battle tank. 


Gunnery scores in live' firing. 


N/A 


Simulator crews scored 4-10% 
better in 2 of 4 conditions. 
No signifk»nt difference in 
remaining 2 oonditbns. 


Magee 
(1984b) 


Tank 

commandar 
training. 
Canadian Army 


Training using gunneiy 
simulator (N-20) 
varsus oonvantional 
instnjctk)n (N « 16) for 
Laopard battle tank. 


Same as above. 


N/A 


No significant differences 
between simulator and 
conventionally trained crews. 


Same as 
above. 


Fixad wino 

flight. 

U.S./USAF 


SImuldtor trainina with 
(N-14) and without 
(N-14) out-of*th6- 
window visual cuSv^ for 
C-130 pibting. 


Instructor ratinos of students 
in aircraft. 

Amount of instmctor assistance 
given. 

Deviations from desired 
performance parameters. 

1 


N/A 


Visuallv cued arouo reached 
proficiency with 25-30% fewer 
sorties. 

About 10% of visually cued 
group required instructor input 
oonpared to 40-50% of non- 
visually cued group. 


Nuilmeyer & 

Rockway 

(1984) 



a 

Reported but restricted information. 



Table 1. Empirical Studies of Training In TTCP Countries (continued) 



Type Of 
Training 


Comparison 


Measures of Effect 


{measures of Cost 


Findings 


Study 


Fixed wing 
flight; mission 
qualification. 
U.S.AJSAF 


Simulator training with 
(N- 14) and without 
(N-12)out-of-the- 
window visual cues for 
(M 30 piloting. 


Required sortie repetitions to 
criterion. 

Profidency ratings. 


N/A 


Visually cued group required 
33% fewer repetitbns to meet 
criterion. 

Mean proficiency ratirtgs 
^ignifk:antly higher for 
visually cued group. 


Nullmeyer & 

Rockway 

(1984) 


Fixed wing 
flight; missbn 
qualification. 
U.S.AJSAF 


Simulating trainir)g with 
(N« 12) and without 
(N«11)out-cf-the- 
window visual cues for 
(M30pibting. 


Required sortie repetitions to 
criterion. 

Proficiency ratings. 


N/A 


Visually cued group required 
32-63% fewer repetitk>ns to 
meet criterion. 

Mean profbiency ratings 6-17% 
higher for visually cued group. 
Twice as many pik>ts in 
visually cued group met 
criterk)n on first attempt. 


Same as 
above. 


Avionics 

maintenance. 

U.S./USAF 


Training using simulator 
(N« 21) versus actual 
equipment (N«22)for 
6883 Converter Fl^ht 
Control Test Statbn. 


End-of-block knowledge test 
(paper-and-pendl). 
Equipment operatk>n test 
(hands-on). 

Equipment troubleshooting 
test (hands*on). 


Acquisitbn costs. 
Life-cycle costs. 


Simulator group scored 1d% 
higher in troubleshooting. 
No other difference in 
performance speed or accuracy. 
TCR-0.27(acquisitk>n) 
TCR-0.03 (life-cycle) 


Pieper, 

Richardson, 

Hannon, 

Keller, & 

Massey 

(1984) 


Automotive 

maintenance. 

U.S./Army 


Training using slide- 
based simulator with 
34)nfX>dule(N«20) 
versus conventional 
lecture and actual 
equipment (Cummins 
NHC-2S0diesel 
engine) (N « 20). 


In maintenance of ^ual 
equipment: 

Time to complete tasks. 
Amount of instructor interventbn 
required. 

Student attitudes. 
Instructor attitudes. 


N/A 


No dffference in number of 
correctly performed steps. 
39% more time and 40% more 
interventbns required in 
simulator group. 


linger, 
Swezey, 
Hays, & 
Mirabella 
(1984) 
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Table 1. Empirical Studies of Training in TTCP Countries (continued) 



Type of 
Training 


Comparison 


Measures of Effect 


Measures of Cost 


Findings 


Study 


Automotiva 

maintananca. 

U.S./Army 


Training using slida- 
based simulator with 
3-Dmoduia(N -21) 
varsua conventional 
lecture and actual 
equipment (Cummins 
NHC-2S0die8ei 
engine (N- 20). 


In maintenance of actual 
equipment: 

Time to complete tasks. 
Amount of instructor 
intervention required. 
Student attitudes. 
Instructor attitudes. 


N/A 


13% fewer correctly performed 
steps, 92% more time, and 
104/5 more required inter- 
ventions in simulator group. 


Unger, 
Swezey, 
Hays, & 
Mirabella 
(1984) 


Samaas 
abova. 


Training using video- 
disc-based simulator 

will 1 ij'tj iiiuuMiD \r% ■ 1 

versus conventional 
lecture and actual 
equipment (starting and 
charging system for 
seV-propellad Howitzer) 
(N-12) 


Same as above. 


N/A 


5% fewer correctly performed 
steps and 16% more time, but 
£47b lewer requirea inierveniions 
in simulator group. 


Same as 
above. 


Samaaa 
abova. 


Training using video- 
disc-baaed simulator 
With 3-D module (N ■ 12) 
versus conventional 
lecture and actual 
equipment (starting and 
charging system for 
self-propelled Howitzer) 
(N-11) 


Same as above. 


N/A 


8% fewer correctly performed 
steps, 35% more time, and 
67% more required interventions 
in simulator group. 


Same as 
above. 


Elactronic 

fnAifitAnAncA 
iiianiiviiaiiitfOi 

U.S./Army 


Training using slide- 

KaaaH ftlfniilirtnr M/fth 
vaow ouiiUMin/i vvhii 

3-0 module (N- 10) 
versus conventional 
lecture and actual 
equipment (l-HAWK 
missile maintananca) 
(N-12) 


Maintenance of actual 

AnijinmAnt 

v\^UI^IIIdll. 

Time to complete tasks. 


N/A 


1 1 % fewer correctly performed 
siepo ana ooto more iime 
required in simulator group. 


Same as 

auuvo. 
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Table 2. Acceptance Testing and Analyses of Training Equipment In TTCP Countries 



Type of 
Training 


System(s) Studied 


Measures of Effect 


Measures of Cost 


Findings 


Study 


Conrmiandand 

Controi. 

UK/Army 


Low-cost, interactive 
graphics (3-D) based 
battlelield engagement 
simulator. 


N/A 


Acquisition costs. 


Less expensive than other 
computer-based systems. 
Might provide facilities useful 
for training. 


Heaton 
(1984) 


Tank Gunnary. 
Canadian Army 


Low-cost, part-task, 
videodisc-based tank 
gunnery trainer for 
Leopard battle tank. 


Tank gunnery proficiency. 


Acquisition costs. 


High face valkiity at k)w cost. 
Possible reductnn in cost by 
a factor of 14. 


Magee& 

Rodden 

(1984) 


Automotiva 
operation and 
maintenance. 
Australian 
Army 


3 light-weight utility 
tnicks (0.7-1.0 tons). 


N/A 


Training days to criterion. 
Number of oonverson 
courses. 

Training aids cost. 


Ratk) of worst case projections 
of training costs to best casr 
projectnns -2.19. 


TSP: 

Project 

Perentie 


Same as 
above. 


2 light cargo trucks 
(1.5 -2.0 tons) 


N/A 


Same as above. 


Ratio of worst case projections 
of training costs to best case 
projectnns « 1 .50. 


Same as 
above. 


Smal amis 
shooting. 
Australian 
Army 


Small arms training 
systems (Lindsay 
Knight Rifle Trainer 
and Superdait 
Projectile Location 
System). 


Shooting proficiency. 


Training time. 

Referenced study on required 
training resources. 
Acquisitbn and devetopment 
costs. 


With equal training effk:iency, 
1st year costs 13% higher, 
subsequent years 35% k)wer. 
With improved effteiency, 
1st year costs 14% lower, 
subsequent years 79% tower. 


TSP: 

Small 

Arms 

Training 

Systems 

(1985) 



Table 2. Acceptance Testing and Analyses of Training Equipment in TTCP Countries (continued) 



Training 


System(s) Studied 


Measures of Effect 


Measures of Cost 


Findings 


Study 


infantry air 

defense 

operator 

training. 

Australian 

Amiy 


3 air defense systems 


Traddng proficiency. 
Interaction with selection 
policy. 

Transfer of sl(ills. 


Training days to criterion. 
Transition training days. 
Number of instructors. 
Acquisition costs. 
Training resources costs 
(e.g., CAI). 

Operating costs per year. 


Without validation capability, 
ratio of worst case to best 
case*---® 

With validation capability, 
ratio of worst case to best 
case- — ^ 


TSP: 

VLLADWS 
Operator 
Training 
(1984) 


Tanic Gunnery. 

Australian 

Army 


7 classes of systems. 
5 turret interaction. 


Gunnery proficiency. 


Ammunitbn required. 


No adequate cost-effectiveness 
data yet available. 
Operator proficiency testing 
needed. 

Operator training should be 
included with new systems. 


Williams 
(1984) 



a 

Reported but restricted information. 



Table 3. Surveys and Analyses of Training 



Sponaor 


Area Reviewed 


Rndings 


Study 


TTCP-UTP-2 


Cost-^ectiveness of military 
training using fNght simulators, 
computer-based instruction, 
and maintenance simulators. 


Simulators and actual equipment are equally effective. 
Savings (compared to actual equipment) from: 
Flight simulators (2-year amortization): 

30-60% of acquisition costs: 

12% of operating costs; 

65% of Kfe-cyde costs. 
Computer-based instruction: 

30% of student time. 
Maintenance simulators (4-year amortization): 

20-50% student time; 

20-60% acquisition costs; 

50% operating costs; 

40% fife-cyde costs. 


Chatelier, 
Harvey, & 
Orlansky 
(1982) 


UK/Navy 


Training simulators for 
submarine command teams. 


Simulators are expected to: 
raduce instructors' workload; 
generate and prepare training materials; 
improve performance evaluation; 
provide realistic/intelligent opposition; 
aid in development of newtac^. 


Cooka 

Maddrell 

(1984) 


U.S^SDRE 


Operating costs of aircraft 
and flight simulators, 
1980-1981. 


Operating costs range from $116-$170 per hour for all types of simulators, across a 
diversity of simulated missions and/or aircraft. 

Simulator-to-aircraft operating cost ratios remained at about 0.08 from 1976 to 1 981 . 
Concerning 15 Air Force simulators and fixed-wing aircraft: 

Both aircraft and simulator operating costs doubled from 1 976 to 1981 . 

Aircraft operating cost increases are consistent with inflation; only 40% of 

simulator operating cost increases are. 

Simulator use reduced 30% from 1 976 to 1 981 . 


Orlansky, 
Knapp, & 
String 
(1984) 
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Table 3. Surveys and Analyses of Training (continued) 



Sponsor 


Area Reviewed 


Findings 


Study 


Australian Navy 


Pcrt-taslc, low-cost trainers 
userl by TTCP member 
countries. 


Total of 47 trainers listed for UK, New Zealand, and Australia: 
8 for flight training; 

14 for gunnery and smaU arms training; 
14 for maintenance training; 
1 1 for operations training. 


Sayer 
(1984) 


U.S7USAF 


Visual and motion simulation 
for flight training. 


Of 26 ' tudies of visual simulation, 18 concerned transitbn training. 
Visu J simulatton shows positive transfer across all types of aircraft. 
Most transfer obtained for formation flying and surface attack weapons delivery- 
less transfer obtained for aerobatic arid air combat skills. 
*n none of 10 studies was aircraft performance significantly enhanced by simulator 
training with platform motfon. 


Waag 
(1981) 
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Table 4. Studies of Methodology for Evaluating Training 



Sponsor 


Area Reviewed 


Findings 


Study 


TTCP-UTP-2 


1 

Evaluation of unit (collective) 
training. 


Theoretical plan and practical means for applying evaluation technology in the field. 
Approach keyed to Systems Approach to Training requiring analysis, design, 
development. Implementation, and evaluation. 


Hudson, 
Jans, 
Thornley, & 
Desmond 
(1985) 


U.STDoD 


Costs for military training. 


A nnodei that identtfies and rtructures a list of all cost elements needed to conduct 
life-cyde cost-effectiveness analyses of alternative programs for military training. 
Cost elements are defined under the general headings of Research arnl Development, 
initial investment, and Operating and Support. Applications of the elements to academic 
training, maintenance training, and flight training are Wustrated. 


KnapoA 
OrlansKy 
(1983) 


U.SVNavy 


CufK^nt practices in cost- 
benefit analyses of military 
manpower and training 
research arxi development. 


Taxonomy of generally accepted and widely used techniques and precepts. 
14 elements of cost-benefit analysis are identified and defined. 
These elements are shown to have been applied to different degrees in 1 1 economic 
analyses of Navy manpower, personnel, and training. 


McMichael 
(1985) 
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Table 5. Effectiveness and Cost Data Provided by 
22 Empirical TTCP Studies 



Typeb of Data 


Number of Studies 


Effectiveness Measures 




End cf Course 


22 


Tran<5fer 


20 


On-the-Job 


0 


Student Attrition 


3 


Instructor Attitudc^s 


7 


Student Attitudes 


7 


Cost Data 




Research and Development 


0 


Initial Investment 




Program Development 


0 


Acquisition 


3 


Operating and Support 




Student Pay and Allowance 


0 


instructor ray arxi Allowance 


A 
U 


Main.jnance and Repair 


0 


Program Modification 


0 


Student Time Savings 


8 


Operating 


4 


Life-Cycle Costs (includes all of the above) 


2 



